genius discovery!! This should explain why it complained about not seeing any image.

Yes, I expect this patch to fix this, but haven't tested yet.

Thanks for the fix. Now we can expect a better performance in 7b as well I guess 🚀

I am trying to replicate the previous results with manually specifying the old 1234 seed, but the result is very different every time, so there might be a bug somewhere (not in this pr).

@Green-Sky Try with the last commit.

@monatis thanks, will also have to patch master though.

I thought it was necessary every time the after the system prompt was evaluated.

Question: What animal do you see in the picture?

Model 7B:

I see a large, hairless animal in the picture.

I expected naked mole rat or something.

I thought it was necessary every time the after the system prompt was evaluated.

Yea, maybe, but before this pr it was clearly wrong, since it inserted BOS inbetween USER: and <user prompt here>

@FSSRepo 13b in f16 (temperature = 0.2):

" In the image, I see a hairless mole with long nails."
The hyperparameters Above are most comparable ones to the online demo to my best knowledge.

@FSSRepo 13b in f16 (temperature = 0.2):

I'm using the ggml-model-7b-q4_k.gguf and temperature = 0.1

This fix is wonderful. Now it works fine even with vicuna-13b-v1.5-Q8_0.gguf, instead of the finetuned llava model.

With the naked mole rat image above, it gives,

The animal in the picture is a naked mole-rat, also known as a hairless mole. It is a small mammal with a long, pointed nose and a hairless body. The naked mole-rat is native to Africa and is known for its unique social behavior and ability to live in large colonies.

And with the DL image, and the json prompt,

{
"class": "DLN",
"DLN": "123456789",
"DOB": "01/01/1980",
"Name": "Iris Cardholder",
"Address": "123 Main St, Anytown, CA 123456",
"EXP": "08/15/2023",
"ISS": "CA",
"SEX": "F",
"HGT": "5'6\"",
"WGT": "125 lbs",
"EYES": "Brown",
"HAIR": "Black",
"DONOR": "N/A"
}

Now it works fine even with vicuna-13b-v1.5-Q8_0.gguf, instead of the finetuned llava model.

how. @haotian-liu how much does the projection matrix and the finetuned clip vit contribute in comparison to the llm ?

CLIP itself is not finetuned in LLaVA. I think the model @jxy is referring to is this model which undergoes further finetuning for insstruction following to become LLaVA V1.5 to my knowledge.

Only the LLM part is finetuned, which is bridged to frozen CLIP layers via a two-layer MLP (multimodal projector) during pretraining.

CLIP itself is not finetuned in LLaVA. I think the model @jxy is referring to is this model which undergoes further finetuning for insstruction following to become LLaVA V1.5 to my knowledge.

I used the actual vicuna v1.5, from https://huggingface.co/TheBloke/vicuna-13B-v1.5-GGUF/resolve/main/vicuna-13b-v1.5.Q8_0.gguf

You may try it yourself. I also tried other 13B models, they all kind of works.

Fascinating that the embedding projection matrix does so much by itself.

@ggerganov Cool! Thanks for identifying and fixing this. The model now works much better.

@Green-Sky

After the first-stage pretraining, LLaVA can actually answer some basic questions quite well, even following the format instructions (Table 6 in llava-1.5 paper).

But there are some tasks that it is not yet good at doing, like reasoning about the relationship between the objects in the scene. For these tasks, it would probably require the finetuned LLM as well.